Software, such as programs or applications, is tested many times before being released to the public to ensure proper operation. The most common test performed on software is a simple text to verify that some functionality is operable. For example, after each substantial revision in the development to determine if the changes in the new version might have detrimentally affected the operation of the software due to unanticipated conflicts or errors. If a problem, such as an error or exception occurring during the execution of a scenario, is found during a test, software testers may utilize a number of testing tools to evaluate the step-by-step performance of software and to identify the source of the problem.
Software testing requires the tester to develop a test in which the tester identifies what operational test scenario the target software should perform, i.e. the defined the sequence of actions the software will take during the test. The software testing process typically requires the tester to write a test script that, at a minimum, identifies the software to be tested and the test scenario to be tested. Additionally, the test script typically configures the computer and target software to a specific starting point and directs the test results to a specific location.
In addition to this, test scripts are typically testing tool specific in that each testing tool has its own format for defining a scenario. Thus, in order to test a particular scenario with multiple tools, as may occur when the source of problem is difficult to identify, a tester must have knowledge of the specific scenario definition requirements for each testing tool and must recreate the test scenario for each tool.
After it has been created, a test script is then executed, such as by a software testing tool or application, which causes the target application (i.e., the software being tested) to perform the actions identified in the test scenario in the test script. The results of the test are then inspected by the tester to determine if additional testing is needed, if the test failed to investigate the source of the failure, or if the tested software has passed the test.
Testing software is a tedious process because often a test must be repeated for many different test scenarios. For example, a simple test scenario for an office productivity application may consist of a) opening a document of a first type, b) saving the document as a second type, and c) closing the application. Such a test scenario is designed to find problems in the “Save As” functionality of the application and verify that each file type is supported by the “Save As” function in the application. Another simple test scenario related to the “Save As” functionality may consist of a) opening a document, b) changing the document's name to a predetermined name, and c) saving the document. This test scenario may be used to test various file names to verify that the application does not generate errors when long file names or file names with special characters are used.
Typically, these simple tests would be repeated many times with only slight variations in the test scenario. For instance, in the above examples the first test scenario may be repeated for each file type that the application supports in its “Save As” menu command and the second test scenario may be repeated for a predetermined set of specific file names designed to test the limits of the software's name support.
A tester confronted with the task of performing tests with multiple variations like the examples described above typically would create multiple test scripts, one for each document type or file name to be tested. Alternatively, the tester could create one script that was written to interface with a data file that identified each document type or each file name and caused the test to be run iteratively until the test had been repeated for each entry in the data file.
The drawbacks of these methods of testing different, but similar, scenarios are many. In particular, the separate script method has the drawback that it is very time intensive. The amount of time spent in writing, executing, and then reviewing the output generated by each test script is incredible; more so considering that each test includes completely reconfiguring the test computer each time and that the outputs are often located in separate files which must be separately opened and evaluated. Some of the drawbacks of using a single script to iterate through a separate data file are that the testing tool may not return the test environment to a known state after each iteration or that an error may corrupt the test environment: each case essentially rendering the results of all subsequent iterations valueless or at least suspect.